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AiJSTRACT 



Performance measurements of a lataoase macnine reflect 
not only tne processins power of tne macnine, out aiso tne 
size ann structure of tne database. It is tnerefore useful 
to const rue t databases for performance measurements of data- 
base macnines. Purtnermore, it is useful to utilize 
syntnetic data, suen tnat tne volume of tne reply can be 
predicted for a ^iven query and tne structure and attributes 
of tne database can be varied for intended test queries. 
Conducting measurement studies usine a syntnetic database 
contributes to tne generality of tne results wnen different 
test Queries are employed. A parameterized program is de- 
scribed nerein wnict can be used to generate various 
relations for a syntnetic database. Tne experiences in con- 
structing and using tne database generator are described It 
is suggested tnat given sufficient information on reai-worli 
databases tne venerator may be useful for modelin? tnem as 
well as for creating databases for oencnmaric tests. 
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I. 3SNCEVARKS FOR DATABASE MACHINES 



'a. performance MEASUREr^SNTS 

In coTiparing aatatase nanasement systems an 

Important factor is ttieir performance. One way to compare 
DBMSs is to run specific applications under a variety of 
systems. Sacn system can be 'fine~tunea' to ?ivp tne best 
result. An evaluation based on sucn a metnod is costly and 
time-consuming. Often sucn a method may be infeasible. In 
many cases, a database for tne specific applicaticns may 
not even exist. As a second metnod, an evaluation could be 
made on tne oasis of performance measurements of existin? 
databases. This metnod is less costly and less time-consu- 
ming. However, tne following questions arise. Is tne 
existing database sufficient to support intended applica- 
tions? Are tne applications good for condu ting relative 
performance evaluation of different DBMSs? 

It is impractical to perform sucn direct comparison of 
DB^Ss. Adapting an application to several systems for eval- 
uation purposes is not practical. Evaluation based on 
existing databases is subject to interpretation error. Tne 
increasing number of DBMSs maAes it imperative that some 
metnod is to be devised to do comparative performance 
measurements . 
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E. BENCHMARKING 

Tne concept of a stanaara for measuring performance i; 
not new. Tne standard is usually inown as a oencnmaric. 
after tne mariers used oy surveyors in estaoi suing a common 
reference point for tiieir measurements. Jor example. Mount 
Diablo (a mountain east of San Francisco) is used as tne 
reference point in surveying mucn of Nortnern California due 
to its long-ran^'e visibility. A metnod for measuring 
similar items in reference to a standard is called 
bencnmariing. 

Precedents for bencnmariins exist in measuring tne per- 
formance of computer systems. Tne Gibson-Mix metncd 
measures tne execution time of a specific set of application 
programs for bencnmarsing computer systems. Tne expected 
performance of a system could be computed by cna rac ter i zing 
tne expected worsloal as a mix of jobs from tne standard 
set. 

It is proposed tnat a set of application programs can oe 
devised to measure tne performance of DBMSs. Using tnese 
bencnmart measurements, it will oe possible to compare tne 
performance of various uSMSs. Tne measurements can oe ana- 
lyzed to su??est stren?tns and weaknesses of tne DBMSs. 

C. QUANTITIES TO BE MEASURED 

Tne firenerally accepted performance index for a DB^'^S is 
tne response time. Defining tne response time as tne 
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primary perf ormancs index is me scope of mis rssearcn. 
However, me response time is based on several factors. 
Among tnese factors are tne tine to process tne query, tne 
time to access tne lata, tne time to process lata, and tne 
time to return tne data. For a DBMS running on a mainframe 
computer, tne effects of otner worKioai on tne response 
time must also be considered. 

A measurement of tne response time is more sis-nif i^'ant 
wnen measurements of its components are provided. Some 
simplifyine assumptions may be made. Tne first sucn assump- 
tion is that tne rate of accessing data in tne database is 
constant. Tne second is tnat tne rate of returning pro- 
cessed data is constant. However, tne time involved in tne 
processing of queries and tne time involved in tne proces- 
sine of data may vary ereatly amon<? database operations. It 
order to record the variance of time among tne operations, 
tests must be devised wnicn will indicate tnese components 
for all supported operations. 

Tnis tnesis focuses on measurements of tne response 
time. A development of a system to measure components of 
tne response time is discussed. Tne system involves tne 
seneration of a syntnetic database. Tne system also mea- 
sures tne bencnmariced macnine in usin? tnat database. 






II. BKNCHMARiiNG RjlLATiONAL MIAMS^ M4CHIN.^S 

A. THE BENCHMARKING ENVIRONMENT 

The research done in support of this thesis nas oeen 
perforTied in a complex environment. Tne complexity involves 
multiple machines and multiple operating- systems. 

A Relation Generator (RG) of synthetic relations nas 
been developed using Pascal (i.e.» IBM's Pascai/VS) in a 
multiuser environment (7M/CMS running on ISM 30^2). RC- is 
used in a oaten environment (MVS) on the same machine. Tne 
relations are generated in EBCEIC-cnaracter form. Tney are 
transported to a UNIVAC 11550 via tape. Tne EBCDIC files are 
tnen loaded onto tne nost (i.e., tne ONIVAC computer^ and 
translated by the host into ASCII files. Tnese ASCII files 
are finally loaded into a bacjend database machine (i.e., 
Britton Lee's IDM 500). 

Tne bacKend machine and interface software for tne 1120 
series computers are marseted by the Amperif Corporation of 
Chatswortn, California, as tne RDM 1100. Additional mea- 
surements can be made by bypassing tne part of tne query 
processor that provides terminal support. This is accom- 
plished by communicating directly with tne query processor 
via compiled language statements (i.e., COBOL). This does 
not completely bypass the query processor, because the query 
language is interpreted and cannot oe precompiled. However. 
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me results snow tnat query processini? does not represent a 
sisn if leant portion of tne response time if tne dost wors- 
ioad is iignt. Tne terminal nandler represents also a small 
portion of tne response time. Tnerefore. tne only advantage 
to tne use of compiled programs is tne option of running tne 
process as a oaesground joD. 

B. THE ARCHITECTURE OF THE SYSTEM 

Tne arenitecture of tne system encompasses two major 
areas. Tne first of tnese areas is tne internal arenitec- 
ture of tne IDM 50fe5. Tne second area is tne nost system 
software, i.e., tne user interface wnicn runs on tne nost. 

1 • Macni ne Arenitecture and Va ri oui C onf igu- 
rations 

Tne IDM 500 is made up of several modules connected 
to a common nien-speed ous (See Figure 1). Tne database 
proce ss or is a 6-mnz , Zilog Z-y000 series microprocessor 
wnicn performs tne DBMS functions. Tne coding for tne 
microprocessor is written largely in tne C programming lan- 
guage, along witn some assembly language routines. It 
comprises about 330 t-bytes of macnine code. An optional 
module, tne data base accelerator improves tne system perfor- 
mance by implementing in nign-speed, special-purpose 
nardware some of tne DBMS functions normally performed ov 
tne database processor. 
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Database 



Figure 1 - The IDM Bus Architecture 
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Tas cac 2 ^ is composea of fc4i-oit oynaric ?.A'^ 

Tne basic configuration (at tne oeginnine of tne 
tests) inciudei one-naif iregaDyte of rterriory. Up to six 
megabytes of memory can be supported. During tne testing 
period, configurations of one and two mecracytes nave also 
been used. 

One to four di_SK 223112. iiSIi '’’S-Y instaiiei- 

Sacn controller supports up to four slx-nundred-mega byte , 
nard dislcs. A lipe coiuroller may be installed tn feriii- 
tate bacicing up and loading data. 

Two standard nost interfaces ere available. k 
ISEE-488 byte-wide parallel interface is available for con- 
nection to mainframes and minicomputers. A second interface 
can be used to provide multiple P.S-232 serial ports to 
microcomputers. A special byte/word interface for com- 

munication witn UNIVAC nost computers is supplied by tne 
Amperif Corporation. 

2. Tne Database Organisation 

The IDM software supports tne relational data- 
base model. Data is stored on tne disic in two logical 
levels. Tnese levels are tne system database and tne user 
databases. At tne top level, tne system database contains 
five system tables and tnirteen database tables. Tne system 
tables contain information on nardware configuration, aata- 
bases and current usaee. Tne tnirteen iatabase tables 
comprise tne data dictionary. Tney are used to 
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store infornation aoout reiaiions, attrlDutes, users, am 
security. A list of tne system taoies and tne lataoase 
taoies is piven in Appendix A. 

Altftouf?n access to tne system database is reouirec 
for tne creation of a user database, an existing user data- 
base can be accessed directly, i.e., wiinout coins’ tnroupn 
tne system database. Eacn user database nas botn database 
tables and user tables. Tne database tables are stored 
rfitnin tne user database and may be accessed in tne same 
manner as user tables. 

Tne basic unit of disK access is a 2ir-byte oiocir. 
Wnen a database is created, a space allocation in blocks nay 
be requested. Tnis allocation may be increased if neces- 
sary. Both system tables and database taoies are used by 
tne system to compute physical addresses. 

3. Tne User interface 

Tne user interface is accessed oy invosine an pro- 
cess on tne host. Tnis process is an intera^^tive query 
processor. Tne auery processor parses tne user's queries 
written in the Relational Cuery Lan?uae'e (RGL). RCL is 
Amperif's implementation of Britton-Lee's Intelligent Query 
Language (lOL). Alternatively, queries may ce submitted to 
tne query processor from a compiled COBOL or FORTRAN pro- 
gram. Submitting a compiled program as a oaten job, tne 
user may oynass tne query processor's terminal nanaler. 
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However, me betcn job still depends on me query pro''essor 
for parsing of tee query. 

The Relational Query Language (RQL) proviies opera- 
tions and facilities similar to tnose available on 
relational DH'^Ss currently running on mainfrarre computers 
and larger minicomputers. ROL also allows queries to be 
pre-parsed and stored within a database. These stored com- 
mands limit the time required in the nost for parsing ana 
reduce the time required in the baefcend for the database- 
table looicup. Additional information on RQL nay be found in 
[1 , 2 and 3J . 

Communication with the IDM is via a system process, 
RDMIO. RDMIO supervises communications between user proces- 
ses running on the nost and the hardware interface to the 
ID!^ (See Figure 2). Up to ten users may access tne Riy 
simultaneously from a single UNIVAC nost. 



IB 




Figure 2 - The IDM/User Interface 
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III. THi: BENCHMARKING APPROACH 



A. A MULTI-DIMSNSIONAL PROBLEM 

Creatin? a DencnmarKi ng system poses a protiem witn 
several dimensions. Tne problem can be broien down into two 
major areas. Tnese areas are modeling and measurement. 

I • £xo^i§.!ns. 

Tne modeling problems can be categorized as DBMS- 
dependent and database-dependent. Tne DBMS-dependent 
modeling problems are related to DBMS scnema and syntax. 
Tne database-dependent problems are related to tne cnara''- 
teristics of tne database and tne application to be modeled. 

a. DBMS-dependent Problems 

Tne tnree widely .-cnown database models are tne 
niera rcnica 1 , tne networK, and tne relational. It ras been 
shown that databases and applications based on one of these 

models can be translated to any other model. However, there 

is no accepted oasis for meaningful comparisons of tneir 
performance measurement. As a first step, tests nave been 

performed in support for establishing sucn a basis for DB^Ss 

having the same underlying model, specifically tne relation- 
al model. 

b. Data base-dependent Problems 

Tne database-dependent problems are representa- 
tive of existing databases and the applications i*nlcn are 



used on tnem. t’llstin? databases vary In tne conniexity and 
in tne efficiency in wnicn tney nave been irnpienentea . 
Tnsse varieties are partly cue to tne pnysicai data tnat are 
represented in tne database and partly due to tne profram- 
fners ' aoiiities to construct tne database. Additionally, 
tne applications wnicn use tnese databases also model tne 
pnysicai lata represented as well as tne information re- 
quired of tne database. Tnus , ootn existing databases and 
applications must be modeled. Tne key to an effective and 
general model is creating one wnicn represents common cna- 
racteristics. Tne cnaracteristics of databases and 
applications must be carwfully studied prior to tne design 
of a general and effective model. Tne contrasting nature of 
existing databases and tneir applications present an ex- 
tremely complex modeling problem. 

2 • ^easuremen t P ro b lem s 

DBMS bencnmark measurements, as a standard, may also 
represent a comparison of DBMS performance. Tnis standard 
may be eitner absolute or relative. Absolute measurements 
assume a fixed standard. Relative measurements may provide 
rankings witnin a group of DBMSs. Tne measurement of tne 
response time for relative ranking is our sroai. 

Experiments must oe constructed carefully and tne 
environment must be controlled to provide useable, accurate 
measurements. For example, in performing researcn for tnis 
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tnesis it nas ceen noticed tnat me load on tne nost can 



significantly affect tne response time as seen oy me nser. 
Similarly, tne response time is neaviiy affected oy tne time 
reauirei to return tne data to tne user at tne screen. 
Tnese effects must be minimized in order to obtain measure- 
ments wnicn more accurately reflect tne performance of tne 
DacKend database macnine. Resolution of measurement pro- 
blems is discussed in Section 7.B. 

B. RESOLVING THE 'MODELING PROBLEf^S 

Altnougn tne modeling problems cannot re 'Eliminated, 
steps can be tairen to minimize tne errors introduced by tne 
modeling process. 

1 Fro Diems 

Two assumptions can be made to minimize tne DS'IS- 
lependent modeling errors. Tnese assumptions concern tne 
formal of tne data and tne operations used to access tne 
data . 

The first assumption is tnat all relations are 
stored in tnird normal form (3NF). Tne use of 3NE minimizes 
tne possibility of inconsistent lata. '•'/nile real databases 
do not use 3NF, tnis fact doesn't discourage cur assumption. 
Tne bencnmars is desienei to provide a measurement of ri'-:S 5 ' 
performance. It is not intended to tase into consideration 
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t&e aDiiiiies of inose persons wno will design tne daiacases 
(altnousn ease of use may oe a conslderaiion in sorre 

instances), for tney may not understand tne tneory cf 5NF. 

Tne second assumption to oe made Is tnat tne query 
languages used by tne DBMSs are iogicaiiy equivalent. Ai- 
tnougn differences in syntax do exist, tney generally do not 
affect tne breaitn of available operations. Tberefore, a 
common set of queries can be implemented in tne DBMSs' 
inlivilual syntaxes and provide tne identical logical re- 
sult. Any variations to tnis snould be noted witn bencnrrars 
results. Tne basic set of experiments include selections, 
projections. Joins, updates, insertions and deletions. Ad- 
ditionally experiments snould be performed wnicn test tne 
performance of any peculiar or powerful operations wnicn a 
DBMS may nave in addition to tne standard set. 

2 . Da t abas e-de pen dent Problem^, 

Tne elimination of data base-dependent modeling pro- 
blems involves two fundamental areas. Tne first of tnese 
areas is tne generation of a syntnetic database. Tne gene- 
ration of sucn a database allows tne use of data wnicn is 
eeneraily representative of existin databases, but net 
specifically representative of any one. Tne lesi?n of tne 
syntnetic database's cnaracteristics snould oe broad. Tnis 
ensures tnat it can be adapted to realistically measure tne 
performance of a database with its own cnaracteristics. 
These cnaracteristics induce tne sizes of me relations (in 
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tti9 numoer of lupies and tuple iengtn) in tne datatase am 
the length of a tuple relative to blocK size of tne storage 
inedium . 

The second area involving database oepencency 
involves tne applications running on tne database. A syn- 
thetic worcload is reouired for tne saine reasons as for the 
synthetic database. Tne design of tne synthetic wcricloal 
Should be broad enough to provide enough results to be acie 
to fully simulate different applications. Tne wor.tload is 
designed with two major considerations. Tne first conside- 
ration is support of the basic relational operations 
discussed previously. An additional consideration taires 
into account the varying access patterns of existin? data- 
bases. For example, a given application may repeatedly 
retrieve only one tuple at a time. Anotner will retrieve 
many in one operation. An important characteristic is the 
locality of tne data retrieved by operations. This charac- 
teristic may produce different levels of performance with 
different indexing methods. 

C. THE SYNTHESIZED DATABASE AND WOF.KIOAD 
1 it Svn tn es i zed Da ta 

In determining a set of benchmark measurements, it 
is necessary to obtain tne set wnicn can ce used on a wine 
range of DBMSs. It is also important that this set does not 
favor any DBMS or class of DBMSs. 
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Two approacnes could nave Deen taKen in octainin? 
measurements. One approacn n/ouid te to perform tests on 
existing latanases. ?ne otner approacn is to do 'measure- 
ments on a syntnetic database. Tne latter allows tne 
sreatest flexibility in performing operations on tne data- 
base. This is because tne scnema of a real database rri^rnt 
not provide a suitable structure for performing a test of 
some operations. Tne scnema of a syntnetic database, on tne 
other nand minimizes any bias resulting from designing tne 
tests around a particular database. 

The research for this tnesis is performed in con- 
junction with evaluation of relational database machines. 
However, tne installation nas no relational database^. 
Therefore, any tests on tne CBI^S would nave to be performed 
on either a synthetic database or a database converted from 
another model. Since tne use of syntnetic databases sup- 
ports a more general approacn in bencnmarxing , tne choice 
has been made to generate sucn databases for bencnmarKins' 
tests . 

iJ. Tyi^es of Synthesized Da ta 

Synthesized data snould nave one major characterist- 
ic. Tne types of data snould be broad enough to test tne 
supported operations of different tyoes of fields 
(i.e., values). l‘or example, in tne research performed for 
this tnesis, tne first two attribute values of each relation 
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nave tne same numeric value. However, me first attriDute 
value is storel as an integer and tne second as a '•naracter 
string. One set of tests selects tuples cased on tne inte- 
ger values; a second set of tests selects tne same tuples 
cased on cnaracter values. Response times may oe affecten 
by tne processias- differences related to tne data types. 
Additional differences may result from tne lime requires tc 
format tne data for output. 

3 • §.£!l§ma of S yn^n®,! i zea Oa ta Used 

Tne syntnesized data used for tnis tnesis nas four 
sets of relations. Eacn set nas several relations witn 
different numbers of tuples. Sacn relation in a set nas tne 
same attributes. Tne attributes are similar among tne four 
sets, differing only in number and lengtn in order to pro- 
vide a range of tuple lengtns. Table 1 snows tne range of 
tuple cnaracteri St ics . 

Tne relations are stored in several databases. Two 
databases are used for testing single-relation operations. 
The first database contains all of tne relations used in 
single relation testing. The second database contains rela- 
tions wnose tuples are of bytes and 'dQVi bytes. Tnis 
database uses compressed fields for strings (i.e., trailing 
blan.^s are dropped) . Several databases are used to provide 
relations for testing join operations. For testing, it is 
desirable to spread tne join operations over tne two disK’S 
in tne system. A full implementation of tnis desirable 



26 






■> 



, - -iTtTItllfKMHir'' 






Table 1. Tne Relation Cnaracteri sties 



Tuple Len^tns : 
Relation Logical Sizes : 
Relation Pnysical Sizes : 
Attributes : 

Attribute Types : 



1Z0, 2'^az Bvtes 

500, 1000, 2500, 5000, 10000 Tuples 

50 xiiobytes to 20 negabytes 



14 ( for 100 byte tuples', 24 (for 
otner tuple lengtns) 



Sequential Integer, Randon Integer, 
Collated Alpnanurneric , Blocks Sets 
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laiaoase piacexent is not; possible, because me siorat; 
allocation algoritnns prevent us from controlling over tn 
storage location of specific relations. 
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17. GENERATING SYNTHESIZED DATA 



A. A PARAMETERIZED RELATION GENERATOR 

Tae Relation Generator (RG) is a parameterized program 
for ffeneratin? relations for a database. EG prompts tne 
user concerning tne cnaracteristics of a relation. First 
tne us®r is instructed to enter tne relation nam*=' and size 
(i.e., tne number of tuples). Tnen, tne program requests 
lata about eacn attribute. Tne aata euuestec includes 
attribute name, value type (i.e., integer, string, etc.) ari 
distribution of tne attribute values. Tne relations gene- 
rated are stored in ASCII files to simplify transfer between 
systems . 

1 . Ca p a bil l ti es 

RG contains routines to generate sequential nurroers, 
random numbers (eitner uniquely or nonuniqueiy) , and cna- 
racter strings in collated order (See Appendix E). Tne user 
may also specify a file wnicn contains a set of values for 
an attribute to be used in generating attribute values. 
This set is called a 'value-set' and tne file is called a 
va luem se t file* is produced by tne utility program, 
7alue-set Generator (described below). Tne actual ran^e of 
values from tne file to be used for an attribute is called 
tne attribute's domain. Tne user specifies tne number of 
values from tne value-set to oe included in tne attribute's 



lomain. It is not necessary tnat tne domain contain ail tne 
values in tne value-set. RG requires tne user to define tne 
distribution of tne attribute values. Tne distribution is 
either in discrete blocKs or random or botn. \ disrrete 
distribution in wnicn tne attribute values are randomly dis- 
tributed may be created oy sorting a relation containing 
discrete oiocss on a random number attribute. 

2 • Ine D ev elopme nt 

a. Tne Development Environment 

RG is written in IDM Pascai/VS, runnin? under 
tne 7M/CMS operating system. V!^/CMS is an interactive, 
multiuser operating system. Seacause of operating system 
limitations, RG has been converted to a IVS (baton) process. 
Standard Pascal syntax nas been utilized as mucn as possi- 
ble. Pascal/VS extensions to tne language nave been used. 
Additionally, some of tne file descriptor information is 
specific to tne operating systems. 

b. Tne Development Process 

Tne first step in tne development of tne system 
is • tne drafting' of a modular framewors. Persons are then 
assigned to develop tne different modules of tne pregram. 
Tne different modules include tne main program, tne main 
generator module and tne individual value-type generator 
modules. Tne individual modules produce specific types of 
values for tne attributes. 
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Tne system nas Deen leveioped usins rroaern sort- 
ware engineering tecnniques. The different modules nave 
been debug^en separately. Program narnesses, wrucn contain 
no logic except to invoice a procedure, nave been used to 
test procedures and su bproceiures . Module stubs, wnicn 
simulate tne actions usually performed by procedures, nave 
been usea in place of tne procedures to test tne main pro- 
gram and tne main generator module. Once debugged, tne 
modules nave been integrated witn tne main program. 

Tne responsibility for generating relations nas 
been assigned to one person. Additional development or tne 
system involved several items in addition to debugging. A 
utility to venerate value-set files nas also oeen created. 
Thus, tne otner members of tne team nave been freed to worg 
on other phases of tne project, 
c. Design Problems 

Two major problems have oeen encountered in the 
preparation of RG. Tne first problem is tne sice of tne 
relations to be generated. In tne original RG design, all 
of tne liniel lists of attribute values reside in tne prima- 
ry memory simultaneously. Tne size of tne largest relation 
that nas been generated is twenty megabytes. This requires 
twenty megabytes of tne virtual memory space just to store 
tne contents of the lists. Additional space would be re- 
quired for tne program and tne overhead associated witn 
liniced lists (i.e., pointers to memory locations). Tnis 
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exceeds the virtual memory space available to a single user 
under V'l/C^S. 



Tnis problem nas been partially solved by acces- 
sing sequential files as a substitute for tne lin<ea lists, 
mere fore only one list of attribute values at a tire is 
stored in tne primary memory. However, a iinxed list of 
some of tne longer attributes venerated requires over two 
megabytes of memory just for tne data, witnout considering 
tne space required for pointers. 

Tne second problem concerns tne transportation 
of tne files of eenerated relations to ar.otner system. 
Under tne VM/CMS system at tne Naval Postgraduate Scnool, 
eacn user is allowed a limited amount of file space. Tnis 
amount is mucn too small to nold most of tne relatiors gene- 
rated. Additional space is available on a temporary (i.e., 
one-day) basis. Also important is tne fact tnat wniie 
7!^/CN!S files can be offloaded to tape, tney are stored on 
tape in a non-standard format. Tnere is no utility program 
to transfer VM/CMS files to tape in standard format. Tnere 
is also no utility program to excnange files between tne 
tapes of V'1/C'^S format and tne tapes of M7S format. 

It is apparent tnat VM/CMS is not tne ideal en- 
vironment in wMch to run tne system. Tnerefore. it nas 
been necessary to convert tne system to run in tn= *^7? en- 
vironment. Tne M7S system writes tapes in tne standard 



format. It also allows tfte user to Have a muci*. larger 
virtual memory space. In retrospect, it mases sense to 
ievelop ttie system in an interactive system (i.e., V’^/C'^S ^ . 
Fast turnaround contritutes to faster proisjram deveioprent, 
and tne interactive environment maites deouggin^^ easier. 

B. A MATBIX OF R 3 LATI 0 NS 

Tne relations generated by RG are designed to support 
experiments over a ran?e of relation sizes and cnaracteris- 
tics. Tnese sizes and cna racteris tics are selected to allow 
maximum flexibility in pursuing experiments witn a minimal 
number of relations in tne test database. Tne parameters 
discussed below are tnose of tne relations produced in sup- 
port of tne bencnmaritine . 

1 • Sta nda r d Temp la t es 

Ail of tne relations are cnarac teri zed by tne same 
eeneral template. Tnis template is snown in Table 2 . Four 
specific templates are derived from tne general one. Tnese 
templates correspond to tne four tuple lengtns used for 
testing (i.e., 100 bytes, 200 bytes, 1000 bytes and 2000 
bytes). Sacn template is used to generate tne relations of 
various sizes (b 00 - 10,000 tuples). Tnus most of tne tests 
can be run on many relations by cnan^ins only tne relation 
name (or tne values of tne range variable) in tne queries. 
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Table 2. 



Sey 


- a seiiuential numoer to oe storea as an in- 
teffer field 


'^irror 


- a sequential numoer (same as xev ) to ce 
stored as a cnaracter strins 


Ranlom 


- a random numoer to oe stored as an integer 
field 


5 | 

Ranlom Unique 


- a unique random numoer to oe stored as an 
integer field 


Co iiatei 


- a cnaracter string to oe stored in aipnaoetic or 
der 


Letter 


- a random aipnaceticai letter 


4 

Sets 


- olocKs of values from value-set files. 


=5^ not used 


in some templates 


4 multiple 


attrioutes dependinff on tne tuple len^tn 
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H • Flexi Di l ii Y 

The relations are lesi,snel to provide fietioility in 
testing. Ideally the tests to be performed are .-cnown before 
desi^nin^ the relations. However, the results from some or 
tne tests may suggest a need for additional tests wnicn nave 
not been previously considered. Accordingly, the relations 
are designed to allow the design of additional tests wltnout 
generating more relations. 

C. TH3 GENERATING PROCEDURE 

The generating procedure consists of tnree pnases. Tne 
first phase consists of desiffnin? experiments and the rela- 
tions to be used in tnose experiments. After tne relations 
have been desiened, they must be created and transported to 
tne testing environment. 

Generating relations is a simple process. First 7G, is 
used to generate any necessary value-set files. Then, RG is 
used to generate relations. HG nas been expanded to produce 
a description file. This file contains the attribute names 
and characteristics of tne attribute values in tne relation. 
Tne description lists both tne format of the e-enerated file 
and tne format of tne relation as it is to be stored in tne 
da ta base . 

1 Generator System 

Tne generator system consists of two major programs, 
the Relation Generator (RG) and the 7aiue-set Generator 
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(V3). Oiner programs ancL cteougging alas .Tiay ce necessary, 
lepencLinff on me envi ronfnent (s ) ia wPicP tiie system is 
impiemenieci. 

a. Tpe Relation Generator (RG) 

RG creates a relation file based on input from 
tne user. It consists of four types of nodules: tne main 
program, tne main generator module, tne individual generato’^ 
modules, and tne collating module. 

Tne 'lain Module - Tne main RG module contains 
very simple logic. EG prompts tne user for tne cnaracteri- 
stics of the relation being generated. First, tne name and 
size (in tuples) of tne relation is requested. Tnen. tne 
user is asJced to determine tne characteristics of tne first 
attribute. Tne attribute cnaracteristics are collected in 
an attribute record (See Table ^). After tne nodule obtains 
tne necessary attribute characteristics, it invoKes tne main 
generator module. 

The main eenerator module, as explned intne next 
section produces linxed lists of attribute values and re- 
turns to tne main RG module. RG tnen invoxes tne collate 
module which is detailed in tne sequel. The collate module 
produces tuples by concatenating sets of attribute values. 
After tne relation nas been generated, tne user is given tne 
option of generating another relation or ending the process. 

Tne Main Generator Module - The rain generator 
module is invosed to produce eacn set of attribute values. 
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Taels. Tiie Sce.err'a of an Attrieute Recori 



Attribute Name 
Attribute Type 
String Lene:tn 
Lower Bound 

Upper Bound 
Generate Mode 
Value Set Name 
Relative Proportions 

Seed 



- assigned attribute name 

- data type of attribute values 

- used for string types 

- first sequential integer and lo- 
wer bound for random integers 

- upper bound on random integers 

- data-type distribution 

- value-set file name 

- discrete distribution specifi- 
cation 

- random integers 



37 



Tns cnaracieristics of an atincute are passea lo ine roauie 
in an attribute record. Using ttiis record, tne main rroduis 
invoices one of several individual generator modules, depen- 
line on tne cna ract eris t ics of tne attrioute. Tne 
individual generator module produces a lintel list of attri- 
bute values witn tne aeslrea type and distribution, and 
returns tne list to tne main generator module. Tne main 
eenerator module opens a sequential file, writes tne attri- 
bute values into tne file, closes tne file, and returns to 
tne main RG module. There are tnerefore several seen flies. 
>cnown as attribute files. 

Collate Module - Tne collate module acts as a 
collator. It pnysically concatenates strings of attribute 
values to form a tuple. It is invoked to assimilate all tne 
attribute values in tne attribute files into a file of tne 
relation. Information describing tne attributes is passed 
to tne collator as an array of attribute records. Tne 
collator first opens tne relation file, and all tne attri- 
bute files. Tne relation is generated a tuple at a time. 
One attribute value from eacn file is read. The values are 
concatenated to produce a tuple. Tne tuple is tnen written 
to the relation file. Tne collator repeats tnis process 
until ail tne tuples nave been produced. 

b. The Value -set Generator (VG) 

Tne Value-set Generator (VG) is a simple utility 
for setting up vaiue-set files for .RG. VG asics for tne name 
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anl size (i.e., tne number of values) of tne veive-sei file 
to be created. Tfte values are entered individually and 
stored as strings in a random-access file for use ty ?.& . 

Converting tne program to run in tne baton envi- 
ronment involves several tasxs. Tnese are tne conversion of 
interactive programs to baton programs, tne submission of 
jobs to tne baton system, and development of tne additional 
statements required to use of tne bated file system. AI- 
tnougn tne programs nad already been debugged in tne 7i^/cyS 
environment, extensive lebue^in? nas been necessary after 
conversion to '1VS. 

Conversion of programs from \TM/CMS to MV.S is 
not a simple process. A virtual carl deem is created in a 
7M/CMS file wnicn contains tne source decic, tne input data 
and tne file data required by tne system. Tnis file is 
submitted to tne baton aueue. Tne input for RG (i.e., tne 
user's replies) are in tne carl lecK witn tne program. 

Altnougn it is not necessary, tne source code 
wnicn generated tne instructions to tne user for tne input 
nas been removed for tne M7S versions. Tne '/id/CMS version 
nas been modified to create a file wnicn contains tne user's 
responses to tne program's prompts. 

Differences between tne baten and interactive 
systems caused tne diffi''ulty in program conversion. Tne 
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Datcti system, MVS, requires rnucn more in me way cf file 
parameter specifications, and is rnucn less for^ivinff wnen 
error conditions exist. Tnere are some error conditions 
wnicn tne user can not foresee. For example, tne systfim nay 
initially allocate space for a relation file on a volume 
wnicn does not nave enougn free space to cover secondary 
allocations. 'Vnen tnis happens tne prosrram is cDorted. 
However, it is not possible for tne user to specify a parti- 
cular dism (i.e., one witn sufficient space) for file 
storage. For the two largest relation files (fifteen and 
twenty megabytes), it nas been necessary to write each of 
tne relations into two separate files on tne oaten system. 
The two files were then combined wnen they loaded into tne 
da ta base . 

3 • Transporting the He la t ions to tne Test bed 

a. Transporting tne Eata to tne cost 

Tne transportation of tne relations to tne nost 
is a two-step process. The first step is tne transfer of 
tne relation files from tne MVS secondary storage to tape. 
A system utility is used to accomplish this. The tapes are 
tnen transported to tne nost, tne UNIVAC 11150, and a similar 
utility program is used to load data into tne nost secondary 
storage. Tne nost utility program translates tne 3BCDIC 
tape files into ASCII disK files. 

b. Loading Eata Into tne Bacicend 

Tne relations are loaded into tne oacsend using 
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a vend or”suppiiecL utility called a translator. Tnis utility 
prompts tne user for information about tne source file, tne 
target database, and tne target relation 

Tne translator utility may be run interactively 
or wltn file input. Tne database into wnicn tne relation is 
to be loaded must already exist. Tne relation into wnicn 
lata is loaded may or mav not exist. Database name, nost 
file name, and relation name must be supplied. Additional- 
ly, for eacn attribute tne attribute name, lengtn of source 
(in ASCII cnaracters), and type of value to oe stored in tne 
database must be supplied. 
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V . GENERATING TEST PROGRAf^S 



A. TEE TEST PLAN 

The general test plan calls for several different types 
of experiments. Among taese are experiments involving only 
one relation (i.e., selections and projections) and experi- 
ments involving more tnan one dataoase (i.e., joins'. 

1 I nvo Ivi ng a Single Relation 

The selection and projection experiments are de- 
signed to measure tne system's performance in retrieving 
data from a single relation. Tne response times measured 
are the sum of four vanaoles: tne time to process a uuery, 
tne time to access tne data, tne time to process tne data, 
and the time to return tne data. Tne time to process tne 
query is defined as tne time to parse tne query. 3y care- 
fully constructing sets of experiments, tnese variacies can 
be estimated. 

Since the time to process a query is so small, it 
may be ignored or combined with overhead for most experi- 
ments. Tor experiments where it is significant, tne 
query-proccessing time is minimized to prevent it from domi- 
nating tne time measurement. resulting in a loss of 
precision. Tne RDM 1100 allows tne parse tree of a query to 
be stored in tne database. This capability allows tne 
replacement of the processing time, which is dependent on 
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tne nost, witn me aaia access ilTie, wnicn is dependent onir 
on me oacicend. Tne additional data access time is tne tire 
to access tne comTiana in storage. Tnis is tne same for 
ail stored commands. 

Tne iar£*est variables are tne easiest to measure 
witn precision. Tnerefore» tney are measured first ana tnen 
eliminated to measure tne smaller variables. 

Tne largest variables are licely to be tnose repre- 
senting tne time to access, process and return data. Tnese 
can be measured witn simple retrieve commands. A time 
measurement of a retrieve wnicn returns all tne attribute 
values of tne tuples in a relation includes tne times of ail 
of tne four variables. However, a time measurement usin.? an 
aesre?ate function (e.?., count, wnicn returns a single 
count of tne tuples meeting tne qualifications of tne Query) 
eliminates tne time to return tne data. Tnus tnis function 
can be used effectively to measure tne time to access and 
process tne data (tuples), i.e., two of tne four variables. 

Furtner, an assumption is made tnat for simple com- 
mands tne processor can process data at a rate wnich is 
faster tnan tne rate tnat data can be orougnt into tne 
memory for processing. Tnis allows tne processing time to 
be ignored. Therefore, tne measurements reduce to a measure 
of tne access time. 
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Having quaniit'ied itie lar?er variaoiss, me tirre to 
process lata may be investigated. It nas been assunec mat 
tne processin? time is not sis-nificant for simple conmands. 
However, if tne commands are made more complex, tnen tne 
processing time is expected to increase. Vi tn a sufficient- 
ly complex command wnicn involves a small data-access time, 
tne processing time may become significant. Tnerefore, 
experiments are conducted wnicn minimize data access tut 
vary in complexity. It is of interest to determine wnen or 
if tne processing' time becomes measureable and significant. 

It is expected tnat projections operations will 
increase tne processing time. Tnerefore, several tests are 
appropriate for testing projections. Tne first set of tests 
measure tne effect of projections on tne processing time. 
Tne second set cnecKs to see if tne processing tine is 
affected py tne type(s) of attribute values projected (i.e., 
integer, string). Tne tnird set of tests measures tne 
performance of a projection on all of tne attributes versus 
a simple 'retrieve all' command. 

After tne time basic variables nave been estimated, 
otner performance factors are investigated. Tne use of 
indices can reduce access time. Ey reducing tne amount of 
lata brougnt into tne memory, tne processing time is aiso 
reduced. However, tne processing time will be increased due 
to index access and searcn. Tnerefore, for some relations, 
tn? use of indices may increase tne response time. Indexine 
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requires a specific set of tests to rneasure its performance 
in various situations. Tne use of Uifferent types of in- 
dices (i.e.f clustered, non-cinstered, multiple 'teys, et''.’> 
must also oe investigated. A.n expected factor in index 
performance is tne ratio of tne index size (in dIocks of 
storage) to tnat of tne relation. 

String compression (removal of trailing- spaces' is a 
factor wnicn can affect tne processing time, tne access tine 
and tne return time. Tne use of compression can reduce 
blocK storage dramatically. Tnis, in turn, redu''es tne 
access time. However, it may require more time to process a 
compressed string versus a non-compressed one, if processini^ 
requires expansion of tne compressed attrioute. If expan- 
sion is not required for processing, tnen tne nost may nave 
to expand it for proper formatting. How expensive fin time^ 
is tnis? Does tnis compensate for tne reduction in tne 
responnse time resulting from returning a smaller amount of 
data (tne compressed strin?) to tne nost?. 

Otner performance factors may be examined eitner 
individually or witnin otner test pro''edures. An example of 
tnis is tne use of different types of attributes (i.e.. 
integer versus string). A complete series of tests can t?“ 
developed to test tnis issue in detail. However, it is also 
appropriate to investigate tnis area in conjunction wltn 
processing time and projections. 
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2 • I nvo ivl !^ore Tnan One He iaii.cn 

Operations involving more tnan one relation fi.e., 
joins) are affectea by tns same time variables as tncse 
involving only a single relation. Initial testing snouii 
involve only two relations. 

It is expectei tnat tne access time will become 
dominant for Join operations. Tnis is because tne same data 
may nave to be accessed repeatedly. f'lemory size nas an 
effect on tne amount of accessing required in a join opera- 
tion, If memory size is lar^e enou?n to allow cotn 
relations to be accessed once ana left in tne memory, tnen 
tne processing time may become significant. In tnis circum- 
stance botn tne access time and tne processing time are 
expected to increase proportionally to tne relation size. 
Tde unknown factor is tne rate at wnicn tne pro''essing time 
increases. However, it may be tnat neitner relation is 
small enous'n to be neld in tne memory for processing. In 
tnis case mucn accessing must be performed. It may also be 
of interest to examine join performance oetween tnese two 
extremes . 

Tne join snould be designed to tace acvantage of any 
size differential between tne two relations. If tne smaller 
relation can be completely neld in tne memory, tnen it can 
be accessed once and brousnt into tne memory. Tne larger 
relation can also be accessed just once as it is orou^nt 
into tne memory as a stream. If, on tne otner nard, tne 



larger relation is orougnt into tne memory, it must ce 
broufftit into tne memory a portion at a time. Tne smaller 
relation may nave to c>e reaccessel for eann portion of tne 
larger relation. 

It is Important to examine tne performance of joins 
botn witn and. witnout selection. In performing tnese tests, 
tne strategy of tne operations snould oe examined carefully. 
Tne selection snouii oe performed, before tne actual Join 
operation to minimize tne volume of data being joined. 

Anotner area of Interest is tne effect of index 
usaee on Joins. Performance nere is expected to improve as 
indicated by tne single relation index experiments. However 
tne specific results may suggest tne efficiency witn wnicn 
tne Join operation nas been implemented. 

If inequality Joins nave been impiementen , perfor- 
mance testing snould oe conducted using tnem. If tney nave 
not been implemented, it may be valuable to snow if, and 
witn wnat difficulty, tney can be simulated. 

Having experimented tne Join operations involving 
two relations, experiments operations snould be conducted 
using larger numbers of relations in one Join operation. £y 
invest igat ing tne performance on multiple join relations, it 
may be possible to isolate a fixed overnead for all tne 
initial Joins. 
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3* i Test Plan 

A general test plan saouia ce aevaloped refcre any 
of tne experiments are designed. It snould oe fiexicie to 
enable testing to follow different patns of oiscovery. It 
Is expected tnat tne results of some experiments may suggest 
other experiments. Time must be alloted for tne expansion 
of any test set. 

However, it must also ensure tnat tne a sufficient 
range of data is obtained. Tne tests must cover tne univer- 
sal operations (i.e., tnose expected of any ) . Amonf 
the universal operations, icnown bottlenecis and creatcpoints 
are specifically tested. It snould also investigate any 
specific strengths, weasnesses or idiosyncrasies of tne 
DBMS . 

B. MBASUP.SMENT TOOLS 

Tne response-time measurements in tnese experiments were 
tafeen from the bacKend-macni ne clocK. This clocit has a 
resolution of 1/60 second and an accuracy within 1/60-tn of 
a second. The response time of the bacKenn machine on small 
relations is dominated by communications overnead. Tne 
minimum response time is about one second. So, of tne tests 
conducted, tne 1/30-second interval is sufficiently 
accurate . 
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However 



« if tne overneai can oe reduced., a more precise 
measurinj^ device is required. Most mainframe operating' 
systems provide a cloctc witn a resolution in microseconds. 
Tnis is not avaiiaole in tne DacKend macnine. 

C. OUHRI SCRIPTS VERSUS PROGRAMS 

Two metnods exist for performing oencnmarx experiments. 
Tnese metnods involve tne use of query scripts and programs. 
Tne first of tnese simulates an Interactive session acces- 
sing tne iataoase. Tne actual terminal input is prepared 
anead of time and stored in a 'run-stream' file, «cnown as a 
query script. Tne nost operating system can ce instructed 
to oDtain its input from a file instead of via tne terminal. 
Tnus a series of tests can oe collected toeetner in a 
script. Additionally tne output can be redirected to a 
file, removing tne overnead in communicating witn a 
te rmi nal . 

Tne use of oaten programs involves muen more of tne 
programmer's time in tne development and debugging of tne 
program. Development of oaten programs also represent a 
larger drain on tne nest's resources. TniS factor could 
severely affect testing at many installations. 

Since queries must oe interpreted wnetner tney cone from 
a Dated job or a script, tne use of eaten programming did 
not offer tne advantages of bypassing tne query processor. 
Tnerefore, tnere is some question wnetner or not a eaten 
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proeram would provide superior performance results. Tnis 
question and tne ease of development of query scriots 
suffsest tnat ttie use of query scripts is tne desired retnod. 
If Datcn programming offers a significant performance im- 
provement, additional testing must oe perfornen using eaten 
Jons. Here it would be wise to run a complete oattery of 
tests in tne interactive environment, followed oy a subset 
of tnese tests in tne oaten environment. Tnis subset snouii 
be designed to test areas wnere tne oaten process may nave 
its most impact (i.e., tne data return time). 

D. INTSRPRLTINS THE DATA 

Tne interpretation of data is a very important part of 
tne testing pnase. Tnere are two reasons for tnis. First, 
conclusions cannot be drawn from raw data. Second, Timely 
interpretation enables tne persons conducting tne experi- 
ments to analyze tne results and identify furtner testing. 

A collection of raw data is very nard to interpret. 
Tnerefore, any results obtained snould be ?rapned immediate- 
ly. Grapning tne results immediately allows rapid 
identification of errors and unexpected results. Related 
results snould also be prapnea toeetner. For example, all 
tne results from a query applied to relations of different 
tuple lenetn and relation size snould be erapned toa"etner. 

Once tne raw data is analyzed, tne grapns may be re- 
fined. Tne grapn axes may be varied as appropriate. For 



exanpie, tne response line nay oe grapneci against tne tuple 
iengin, against me relation size (in tuples or me nuncer 
of Dlocffs of me storage space occupiea) and against ms 
quantity of lata returned to me user. 
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i/I. CONCLUSIONS 



A. RESULTS 

Tne results ooiainea from testing several conf i.?ura t ions 
of a relational datanase mactiine nave provided a oasis for 
developing a general set of oencnmaric tests for relational 
database macnines. Tne oencnmaririns tests nave been mostly 
macnine independent. Altnougn a testing metcodoiogy is 
provided herein with enough results on certain configura- 
tions, additional testinsr is necessary. This testine should 
be performed on otner DBMSs, preferaoly with different cna- 
racteristics, to ensure that tne test is complete and not 
macnine-speci fi c . Tne results of testing selection anc 
projection operations are described in [4]. Results from 
performing tests on Join operations are described in [bj . 

^ • 5 ^herai Re sults 

The response time nas been sn wn to be proportional 
to the time required to access tne data. This, in turn, nas 
been shown to be proportional to physical size of tne data- 
base. Hetnods used to reduce tne amount of data to oe 
brougnt into tne memory for processing (sum as indexing and 
string compression) improve tne response time. 

The response time is also proportional to tne amount 
of lata returned to tne user. In tne case of tne RDM , 
the time reouired to return tne lata is tne iar^est 
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component of tne total response time. If tne necessary 
information is ootainel via asi^reeate functions, tne re- 
sponse time is greativ improved. It is not possitie to 
determine now mucd of tne response time is due to tne raci- 
end macnine and now mucn is due to tne dost. However, 
loading tne nost definitely degrades tne response time. An 
analysis of tne response time under various load 'conditions 
in tne nost may lead to a distinction of tne nost response 
time vs. tne baciend response time. 

Tne time required to process queries and tne time 
required to process data in tne memory are relatively small 
for tne RDM lldH. Tnis may not oe true for otner systems. 
Tnerefore, it is imperative tnat tnese areas ce carefully 
examined wnen adapting the proposed tests to systems with 
different architectures. 

The results of tne experiments snow tnat DtMSs do 
nave characteristics wnicn may oe measured. A well-con- 
ceived series of tests can measure an installation's 
performance, and gain an indication of its performance and 
its 'personality.' Tnese tests can oe used to compare DRMSs 
against each otner. For tne DBMS implementor, me tests 
also provide a method of determining poorly implemented 
parts of tne system. 
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2. Resea rcn Results 



Tne eicperimeats wnicn nave teen pert’orrnea nave 
supported two different types of stuay. Tr.e first is tr.e 
actual nieasurenent of tne bacicend macnine's performance 
(albeit, witn lignt load and few configurations). Tne RD'I 
provides a comprenensi ve (altnouen uncomplete) rela- 
tional model wnicn successfully offloads Ei’^S tasKs from tne 
nost . Since evaluation of tne macnine was conducted simul- 
taneously witn tne researcn, tne tasK of evaluating it nas 
been accompl isned . Some areas tnat nave not been fully 
investieated are due to tne lacK of time. Otner areas tnat 
nave net been fully inves toga ten are due to inr-omplete 
implementation. As an example of tnese areas, tne use of ALL 
in a retrieve s is contigent upon tne number of attributes. 
At one point, tne use of ALL on a relation witn a lar?e 
number of attributes results in only an error message. After 
installation of tne accelerator, tne use of ALL naits tne 
command. After tne accelerator is removed, tne problem of 
nalting persists. Anotner deficiency noted nas been tne 
inability to perform an inequality join. 

E. A RELATIONAL SSNCHidARKI NG .'dETHODOLOOr 

Tne proposed set of benenmars tests nas four pnases. 
Tne first pnase consists of preliminary tests designed to 
identify tne best metnod of measuring tne system's response 
time. Tne second pnase involves isolating tne different 
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corrponsnts of trte response tiFie. Tne tniri pnase investi- 
gates tne system response in specific areas. Tne fcurtn 
pnases verifies tne results ootained during tne onases two 
and tnree. 

1 2m Z M£i.iliro meni t;le t no d s 

Most systems nave at least one mecnanism wnicn pro- 
vides a time measurement. Initial testing is designed to 
identify tne one wnicn optimize tne precision obtained ver- 
sus tne ease of ootaining tnat time. Once tne measurement 
metnod nas been cnosen, it is cnecxed to ensure tnat it is 
accurate enoufi*n to provide tne necessary precision. It is 
also necessary to ensure tnat tne overneaa involved in 
retrievinsr tne time does not reduce tne precision of tne 
measurements oeing taien. 

If tne necessary precision is not readily available, 
then tecnni^ues are available to increase tne precision of 
tne results. Tnese tscnniques involve performing an opera- 
tion several times and calculating an average. Tne tecn- 
niques selected must be reviewed for side effects. Tne DJ;MS 
may nave tne capability of internally optimzing performance. 
For example, tne order in wnicn tne queries are submitted to 
tne DBMS may allow tne DBMS cacne memory management to 
reduce disic access. 

In tne case of tne RDM two different metnods 
of measuring time could nave been used. Tne first metnod is 
to obtain a time stamp from tne nost operating system. 
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Aliaough ii i\ay nave proviaed sufficient precision, it nas 
not teen investigated because of tne otner netnocs 
available. Tne second rretnod is a tine^ stamp available from 
tne ID(^. A built-in function supplies an elapsed time 
measurement intervals of one-sixtietn of a second. Tnis 
provides sufficient precision for tne measurements. Since 
tne elapsed time is a sufficient measurement, tne more 
precise measurement nas not been usee. 

'd . Pna se Two - Component isolation 

Once an adequate metnod for measuring time nas been 
verified, it is used to measure tne performance in several 
Specific areas. These areas are tne four components Vkhicn 
are involved in all queries: tne time of process tne uuerv 
(i.e., parse it), tne time to access tne data in tne data- 
base, tne time to process tne data in tne memory, and tne 
time to return tne requested. Tnese components may ce 
considered tne DBMS'S primitive operations. Tnese prim^i- 
tives do not tate advantage of any metnods used to improve 
tne response time of a given query. They me rely measure tne 
performance of tne hardware and software in performing spe- 
cific functions. It nas been stated tnat a performance 
measurement of some aspects of a DDf^S is reaiiy a measure- 
ment of the opera tine system. The operatine system does 
effect DBMS response. However, in tne case of a bacsenc 
machine, this effect is minimal for some operations, 'rfnile 
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tnis issue may oe debated, it is not of interest to t^ie 
user. Tne user is not interested in tne reasons wny e 
system responds poorly. Ke is interested only in tne fact 
tnat a system performs properly and tne fact tnat tne sy- 
stem's performance is better (or worse) tnan tnat of anotner 
system. He is most interested in tne possibility of obtai- 
ning a quiclcer response time on nis application. 

Tne system primitives are measured by a set of 
queries wnicn isolate different aspects of tne response 
time. One set of queries is desiiened to return tne same 
amount of data from relations witn tne same number of tu- 
ples, but naving different tuple sizes. Once a tuple is in 
tne memory, it taKes tne same amount of time to project one 
attribute from a set of 100-byte tuples as from a set of 
2000-byte tuples. Tne difference in tne response time for 
tne two queries is due only to tne time necessary to crins 
tne tuple into tne memory. Tne times required to process 
tne query, to process tne lata and to return tne data are 
tne same. 

Tne second set of queries is designed to measure tne 
time required to return tne lata to tne user. Tnese queries 
return a different amount of data (in bytes) from projection 
operations on tne same number of attributes in me same 
format (i.e., strings, etc.) in relations wnicn are of tne 
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sane pnysicai size. Tnese restriciions assure tnai lue 
access time is tte sane, the processing tine is the sane, 
anl tne query processing time is tne sane. 

The thirl set of queries is lesisned to isolate data 
processing time. In this set, tne queries return tne sane 
amount of lata from relations of the same pnysicai size 
(i.e., identical storage requirements) out having a differ- 
ent number of tuples. This provides a measurement of me 
processing required relative to tne number of tuples pro- 
cessed. The query processine* time, the lata access tine, 
and tne lata return time are tne same. 

The fourth set of queries provides a measurement of 
query processing time. For operations on relations of any 
significant size, tnis is hard to measure. Even on small 
relations, it may not be significant compared to simple 
system overhead. This set of queries is more complex tnan 
the provious sets. Tne queries are constructed to allow tne 
effects of tne time elements (i.e., tne three just measured' 
to be subtracted from tne measurements, leaving only tne 
query processins' time. Considering the difficulty in ob- 
taining a precise measurement of tne query processing time, 
it may not be worthwhile to determine tnis value because of 
its small size. 

The previous discussion indicates that the query 
sets are independent. However, with proper planning tne 
query sets may be combined witn equivalent results. In tne 
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ffrapt shown in Figure 3, one set of experiments provides a 
measurement of lata access times anl lata return times. The 
set also isolates the constant query overheai (whicn 
includes the query processing time). 

Figure 3 represents tne response time of two que- 
ries. One query selects five percent of the tuples am 
returns all of tne attrioute fielas from each tuple. The 
second query is identical except that it sele''ts ten percent 
of tne tuples. The queries are Doth run against relations 
with 10fe'-Dyte tuples. The relations vary in size from P00 
tuples to 10,000 tuples. Point A on the graph represents 
the five percent selection on 10,000 tuples. Point P repre- 
sents tne ten percent selection on 5000 tuples. Since each 
of these queries returns 500 tuples, tne time to return tne 
data is the same. The overhead associated with each query, 
including query processing time, is the same. Therefore, 
the difference De tween tne response times represented oy 
Points A and 3 is tne difference is tne access time and the 
processinsr time of the queries. Point A represents a re- 
trieve on 10,000 tuples, Which is 500 dIocks of disic 
storage. Point B represents a retrieve on 5000 tuples, or 
H50 disc DlocKs. Assuming tnat processing time for these 
queries is insignificant relative to tne access time, fore, 
the difference in the two response times is the time to 
access 250 disK olocics. 
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FIGURE 3 - RESPONSE TIME VS- SIZE OF RETURNED DATA 



The overheaa for ail the queries sno'«n on the ^rapn 
is the same anl is represented oy the conmon intercept cf 
the vertical axis. If the time represented ay Point P is 
adjusted for the overhead and tne time to access gio'^es, 
then tne result is tne time to return 5fc52? ifcJiJ-oyte tuples. 
Therefore, the use of one auery set has identified rates for 
accessing data (in blocks per second) and returning lata (in 
bytes per second) 

3* Phil® Tnree - Sy,stem Response 

After tne time elements nave been measured, a set of 
queries are performed which measure the effect of methods 
used to improve tne system response. An example of tnis is 
the use of indexes. Theoretically, tne use of indexes 
Should improve system performance by decreasing the amount 
of data accessed. However, tne index must be accessed and 
processed. Areas of inter est here involve determining at 
wnat point, if any, does tne use of indexes become impor- 
tant. Therefore, performance on indexed relations is 
measured over a wide range. Wnat type of index (i.e., clu- 
stered or non-clus tered ) provides tne best performance and 
what are the trade-offs? Wnat scope of indices (i.e., one 
attribute, two, or more ) provides tne best pe^'f ormance? 
The latter Question may be one dependent on the appl i''a t ion . 
In testing tne RDM llfe'ii, it has been noted tnat, if the 
index is defined when the relation is being created, then 
the size of a relation with a clustered index is larger then 
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tne size of ine same relation if tne inaex is defined, after 



tne data nas been entered into tne relation. Tnis is be- 
cause tne loading al£-oritnm assumes a normal distribution of 
icev values, wniie tne data is in iey sequence, data loaded 
nas been generated already sorted. 

Additional testing snould be performed to ?et a 
'feel' of tne system. 3y becoming familiar 'Vitn tne sy- 
stem's capabilities, tne testing personnel snould oe able to 
determine interesting lines of experimentation. Areas of 
interest include tne overnead associated »<itn projection 
operations, tne use cf string oompressicn tecnniques , and 
tne efficiency of join operations (in different available 
memory configurations, wnen available). 

-• Pna se Four - Verific at ion 

Tne last pnase taices place after tne otner tests 
nave been reviewed and grapned. Analysis of tne previous 
tests snould provide some meanin.gful results about system 
performance in general, and in particular areas. Tne veri- 
fication pnase serves to perform tests wnicn verify or 
disprove tne analysis of tne previous tests. It also pro- 
vides an opportunity to reoo any tests wnicn appear errone- 
ous or suspicious. In tnis pnase, additional tests may taxe 
advantage of tne flexibility designed into tne syntnetic 
data base . 
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C. SUMMARY 

Investigation of tne performance of several cor.fi«-ura- 
tions of a bactend relational database macnine nas proviiea 
considerable iasi^nt into wnat may be a sound oasis for 
general performance testing on relational l^BMSs. In tnis 
thesis, a metnodoiopy has been laid out and tne initial 
pnases to be taxen in tnat netnodoiogy nave oeen aefined. A 
complete frameworx for subsequent pnases nas net been fully 
developed, out their contents nave been discussed. Vnile 
tne tests described relate tc a specific series of relation- 
al database macnines, tne basic metnoaology may apply to 
relational database macnines. 
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APPENDIX A 



I DM S^s te[p Tables 

“ILiiE Na pl es 

1. Databases - cataiof oi‘ latabases in tne system 

2. DisKS - list of disKS Known to system 

3. Lock - used by IDM for concurrency control 

A. Configure - information about serial ana parallel in- 
terfaces, cnecffpoint interval 

5. DPinstat - information about current activity in tre 
IDM 

DiliOase Tables 

1. Relation - catalog of all objects (relation, view, 
stored command) in tne database 

2. Attribute - catalog of eacn attribute of eacn relation 

3. Indices - catalog of indices tnat exist in tne database 

4. Protect - cataloe of protection information in tne da- 

tabase 

5. Query - stored commands and view 

5. Crossreference - catalog of dependencies among relation- 
views and stored commands 

7. Transact - transaction 10 £ri?in? relation 

B. Users - mapping of user ana group names to user ID 

9. Host_Users - mappinfr from nost ID and user ID to IDM ID 

10. BlocAalloc - catalog of disK blocis 

11. Diss_Usas‘e - database allocation 

12. Baton - temporary transaction logging relation 

13. Descriptions - user definable descriptions 
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APPENDIX B 



Database Generator P_rogram 
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PROCEDURE CHAR, SEC (I^_REC : ATTR.REC; RECAT I CH_S IZE : INTEGER; 
VAR T0P_CF_LIST : C.POINTERI; 

L* GENERATES A LIST OF CHARACTER STRINGS IN ALPHAtiE T ICAL ORDER. 
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Database Generator Program (MVS PascalVS) 
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PROCEDURE INT.SEQLENTIAL UN,REC i ATTR^REC; RELATJON_SI ZE : INTEGER 
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RND_N£R := RANDOM ( 0 ); 

REAL_^UM := (LOW - LOW ★ RND.NBRI + HIGH * RND_NBRi 
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APPENDIX D 



Valueset Generator Program (MVS PascalVS) 
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